Systems and Methods for Automatic Exposure of Images

ABSTRACT

An example method includes capturing, by an image capture device, a first image having a plurality of pixels. Each pixel includes a plurality of channels, and the first image is captured in accordance with first exposure parameters. The method includes determining, by a controller of the image capture device, average pixel intensities for each of the plurality of channels. The method includes determining, by the controller, a weighted average of pixel intensities using the average pixel intensities. The method includes setting, by the controller, a gain that is proportional to a ratio of a desired average pixel intensity relative to the weighted average of pixel intensities. The method includes setting, by the controller, second exposure parameters for a second image based on the gain. The method includes capturing, by the image capture device, the second image in accordance with the second exposure parameters.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/124,205, filed Dec. 16, 2020, which is incorporated herein byreference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Image capture devices, such as cameras, include an array of lightdetectors that sense light in accordance with exposure parameters (e.g.,exposure time, gain, etc.). Forming an image using the array of lightdetectors involves sampling outputs from the light detectors todetermine light intensities received at each point in the array. Amulti-channel image (e.g., an RGB image) may have different lightintensities for different channels (e.g., different color channels).

Lighting conditions in the environment may affect the usability of amulti-channel image for image processing operations such as edgedetection and object detection. For example, one or more channels maybecome overexposed due to lighting conditions that cause a singlechannel to disproportionally affect automatic exposure of the image, andoverexposure of the one or more channels may impede the image processingoperations.

SUMMARY

In a first example, a method is described. The method includescapturing, by an image capture device, a first image having a pluralityof pixels. Each pixel includes a plurality of channels, and the firstimage is captured in accordance with first exposure parameters. Themethod includes determining, by a controller of the image capturedevice, average pixel intensities for each of the plurality of channels.The method includes determining, by the controller, a weighted averageof pixel intensities using the average pixel intensities. The methodincludes setting, by the controller, a gain that is proportional to aratio of a desired average pixel intensity relative to the weightedaverage of pixel intensities. The method includes setting, by thecontroller, second exposure parameters for a second image based on thegain. The method includes capturing, by the image capture device, thesecond image in accordance with the second exposure parameters.

In a second example, image capture device is described. The imagecapture device includes a light detector array and a controller. Thecontroller includes one or more processors, a memory, and programinstructions stored in the memory and executable by the one or moreprocessors to perform functions. The functions include capturing, by animage capture device, a first image having a plurality of pixels. Eachpixel includes a plurality of channels, and the first image is capturedin accordance with first exposure parameters. The functions includedetermining average pixel intensities for each of the plurality ofchannels. The functions include determining a weighted average of pixelintensities using the average pixel intensities. The functions includesetting a gain that is proportional to a ratio of a desired averagepixel intensity relative to the weighted average of pixel intensities.The functions include setting second exposure parameters for a secondimage based on the gain. The functions include capturing the secondimage in accordance with the second exposure parameters.

In a third example, a non-transitory computer readable medium isdescribed. The non-transitory computer readable medium has instructionsstored thereon, that when executed by one or more processors causeperformance of functions. The functions include capturing, by an imagecapture device, a first image having a plurality of pixels. Each pixelincludes a plurality of channels, and the first image is captured inaccordance with first exposure parameters. The functions includedetermining average pixel intensities for each of the plurality ofchannels. The functions include determining a weighted average of pixelintensities using the average pixel intensities. The functions includesetting a gain that is proportional to a ratio of a desired averagepixel intensity relative to the weighted average of pixel intensities.The functions include setting second exposure parameters for a secondimage based on the gain. The functions include capturing the secondimage in accordance with the second exposure parameters.

Other aspects, embodiments, and implementations will become apparent tothose of ordinary skill in the art by reading the following detaileddescription, with reference where appropriate to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a system including an image capture device,according to an example embodiment.

FIG. 2 is a block diagram of another system including an image capturedevice, according to an example embodiment.

FIG. 3 is a block diagram of a simplified method of automatic exposureof an image, according to an example embodiment.

FIG. 4 is an input image, according to an example embodiment.

FIG. 5 is an output image using non-weighted automatic exposure,according to an example embodiment.

FIG. 6 is an output image using channel-weighted automatic exposure,according to an example embodiment.

FIG. 7 is an output image using selective channel-weighted automaticexposure, according to an example embodiment.

FIG. 8A is a portion of the input image, according to an exampleembodiment.

FIG. 8B is a portion of the output image using non-weighted automaticexposure, according to an example embodiment.

FIG. 8C is a portion of the output image using channel-weightedautomatic exposure, according to an example embodiment.

FIG. 8D is a portion of the output image using selectivechannel-weighted automatic exposure, according to an example embodiment.

FIG. 9 is a block diagram of a method, according to an exampleembodiment.

DETAILED DESCRIPTION

Example methods, devices, and systems are described herein. It should beunderstood that the words “example” and “exemplary” are used herein tomean “serving as an example, instance, or illustration.” Any embodimentor feature described herein as being an “example” or “exemplary” is notnecessarily to be construed as preferred or advantageous over otherembodiments or features. Other embodiments can be utilized, and otherchanges can be made, without departing from the scope of the subjectmatter presented herein.

Thus, the example embodiments described herein are not meant to belimiting. Aspects of the present disclosure, as generally describedherein, and illustrated in the figures, can be arranged, substituted,combined, separated, and designed in a wide variety of differentconfigurations, all of which are contemplated herein.

Further, unless context suggests otherwise, the features illustrated ineach of the figures may be used in combination with one another. Thus,the figures should be generally viewed as component aspects of one ormore overall embodiments, with the understanding that not allillustrated features are necessary for each embodiment.

By the term “about” or “substantially” with reference to amounts ormeasurement values described herein, it is meant that the recitedcharacteristic, parameter, or value need not be achieved exactly, butthat deviations or variations, including for example, tolerances,measurement error, measurement accuracy limitations and other factorsknown to those of skill in the art, may occur in amounts that do notpreclude the effect the characteristic was intended to provide.

I. Overview

An image capture device may capture images using one or more exposureparameters that are configured to produce a quality image. For example,a computing device associated with the image capture device maydetermine an average pixel intensity value that is desirable for imagescaptured by the image capture device, and may adjust exposure parametersin order to achieve images with the desired pixel intensities. Forexample, these parameters may include one or more of an aperture,shutter speed, and detector sensitivity (e.g., an InternationalStandards Organization (ISO) rating of the image capture device).Automatically adjusting exposure parameters in this manner is referredto as “automatic exposure,” otherwise referred to as “auto exposure” or“AE.” As used herein, the terms “automatically” or “auto” refer to anoperation performed by, or caused to be performed by, a computing devicewithout human input.

Adjusting the exposure parameters may include determining a gain for howmuch light is used when capturing an image relative to a previous image.For example, one or more of a shutter speed of the image capture deviceor a lens aperture of the image capture device may be adjusted toachieve the gain. However, in examples with multi-channel images (e.g.,RGB images), one channel may reduce the average pixel values and mayresult in an unnecessarily large gain. An overlarge gain may result inone or more channels becoming overexposed, and having clipped pixelswith truncated visual information. As used herein in the context of apixel in an image, the term “clipped” refers to the pixel having anintensity that is above a maximum pixel intensity. This causes the pixelto be set at the maximum intensity by default, and results in a loss ofinformation because an intensity of the pixel relative to other pixelsis lost. Images with clipped pixels may lack detail, and may be lesssuitable for image processing operations, such as edge detection, objectdetection, or text detection operations.

In an example embodiment, a weighted average of pixel intensities isused for adjusting exposure parameters for an image capture device. Theweighted average may include weights associated with each channel, thesummation of which results in an output weighted average of pixels thatmore accurately represents the average pixel intensity and lessens orprevents clipping. For example, a luminosity function can be used to setthe weights in the weighted average, which weights greens above reds,and which weights reds above blues (e.g., WeightedAverage=0.2126*R+0.7152*G+0.0722*B). In this manner, the gain may betailored more closely with perceived brightness in the image and avoidclipped pixels.

In an example embodiment, the weighted average is further refined basedon determining that there is an outlier channel. For example, a bluechannel may have significantly lower pixel intensities than either thered or green channels. A computing device may determine that the bluechannel is an outlier based on comparing average pixel values in theblue channel to average pixel values in the red and green channels anddetermine that the average pixel values differ by a threshold amount(e.g., greater than or equal to 30% of a target pixel intensity for anauto exposed image). If the blue channel is determined to be an outlier,the weighted average can be altered to ignore the blue channelcompletely. For example, in the luminosity function example the weightedaverage would be updated as follows: WeightedAverage=0.2486*R+0.7512*G+0.0*B). This may further reduce the chance ofclipped pixels when exposing an image based on the weighted average.

Within examples, the weighted average in a first image can be used toauto expose a second image that directly follows the first image in asequence of images captured by the image capture device. In this manner,a single image can be used to correct exposures in one or moresubsequent images.

Within examples, determining the weighted average may be performedresponsive to a prompt. For example, an operational condition of theimage capture device or a system of the image capture device may promptthe weighted average for use in setting exposure parameters for theimage capture device. The operational conditions may correspond toscenarios in which lighting causes intensities of one channel to becomedisproportionately large or small relative to the other channels. For anRGB image capture device, this may occur at a certain time of day (e.g.,sunrise or sunset) or in particular settings or locations (e.g., tunnelsor roadways with artificial lighting). In other examples the prompt maycorrespond to determining that one or more channels is overexposed. Inthis manner, an image capture device may adaptively shift from a firstmode of auto exposure to a second mode of auto exposure, so that imageprocessing operations on output images will more likely be effective.

II. Example Systems

FIG. 1 is a block diagram of a system including an image capture device,according to an example embodiment. In particular, FIG. 1 shows a system100 that includes a system controller 102, an image capture device 110,one or more sensors 112, and a plurality of controllable components 114.System controller 102 includes processor(s) 104, a memory 106, andinstructions 108 stored in the memory 106 and executable by theprocessor(s) 104 to perform functions.

The processor(s) 104 can include on or more processors, such as one ormore general-purpose microprocessors and/or one or more special purposemicroprocessors. The one or more processors may include, for instance,an application-specific integrated circuit (ASIC) or afield-programmable gate array (FPGA). Other types of processors,computers, or devices configured to carry out program instructions arecontemplated herein.

The memory 106 may include a computer readable medium, such as anon-transitory computer readable medium, which may include withoutlimitation, read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), non-volatilerandom-access memory (e.g., flash memory), a solid state drive (SSD), ahard disk drive (HDD), a Compact Disc (CD), a Digital Video Disk (DVD),a digital tape, read/write (R/W) CDs, R/W DVDs, etc.

The image capture device 110, described further below with respect toFIG. 2, includes light detectors configured to detect light. The imagecapture device 110 is configured to provide a two-dimensional (2D) arrayof data (e.g., an image) based on outputs of the light detectors. Thesystem controller 102, in turn, may perform operations on the image todetermine characteristics of an environment (e.g., edge detection,object detection, text detection, or the like).

Similarly, the system controller 102 may use outputs from the one ormore sensors 112 to determine characteristics of the system 100 and/orcharacteristics of the environment. For example, the one or more sensors112 may include one or more of a Global Positioning System (GPS), anInertia Measurement Unit (IMU), a light sensor, a time sensor, and othersensors indicative of parameters relevant to the system 100 and/or theenvironment). The image capture device 110 is depicted as separate fromthe one or more sensors 112 for purposes of example, and may beconsidered a sensor in some examples.

Based on characteristics of the system 100 and/or the environmentdetermined by the system controller 102 based on the outputs from theimage capture device 110 and the one or more sensors 112, the systemcontroller 102 may control the controllable components 114 to performone or more actions. For example, the system 100 may correspond to avehicle, in which case the controllable components 114 may include abraking system, a turning system, and/or an accelerating system of theof vehicle, and the system controller 102 may change aspects of thesecontrollable components based on characteristics determined from theimage capture device 110 and/or the one or more sensors 112 (e.g., whenthe system controller 102 controls the vehicle in an autonomous mode).Within examples, the image capture device 110 and the one or moresensors 112 are also controllable by the system controller 102. Forexample, system controller 102 may determine one or more operationalparameters (e.g., a location of the system 100 or a time of day), andcontrol an operation mode of the image capture device 110 based on thedetermined operational parameter.

FIG. 2 is a block diagram of another system including an image capturedevice, according to an example embodiment. In particular, FIG. 1 showsa system 200 that includes an image capture device 202, one or moresensors 214, and a computing device 216. The computing device 216 may beconfigured similarly to the system controller 102 described above withrespect to FIG. 1. The image capture device 202 includes processor(s)206, a memory 208, and instructions 210 stored in the memory 208 andexecutable by the processor(s) 206 to perform functions. The imagecapture device 202 also includes a light detector array 212.

The processor(s) 206 can include on or more processors, such as one ormore general-purpose microprocessors and/or one or more special purposemicroprocessors. The one or more processors may include, for instance,an application-specific integrated circuit (ASIC) or afield-programmable gate array (FPGA). Other types of processors,computers, or devices configured to carry out software instructions arecontemplated herein.

The memory 208 may include a computer readable medium, such as anon-transitory computer readable medium, which may include withoutlimitation, read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), non-volatilerandom-access memory (e.g., flash memory), a solid state drive (SSD), ahard disk drive (HDD), a Compact Disc (CD), a Digital Video Disk (DVD),a digital tape, read/write (R/W) CDs, R/W DVDs, etc.

The light detector array 212 may include a plurality of adjacent lightdetectors configured to provide an output in response to receivinglight. Different light intensities received by a given light detectorresult in different outputs (e.g., voltage and/or current outputs). Forexample, a light detector can be a photoconductor (photoresistor),photovoltaic device (photocell), phototransistor, a photodiode, oranother device capable of producing an output indicative of an intensityof light received by the light detector. Within examples, each lightdetector of the light detector array 212 may correspond to a pixel in anoutput image. Thus, within certain contexts herein, a light detector maybe referred to in terms of its corresponding pixel in an image. Thelight detector array 212 may further include circuitry, such as a powersource and connectors used for providing power to each given lightdetector and thereby driving an output associated with a received lightintensity at the given light detector.

The image capture device 202 is configured to capture images usingoutputs from the light detector array 212. The controller 204 isconfigured to control aspects of image capture, such as exposureparameters of the image capture device 202. For example, the imagecapture device 202 may include a lens and a shutter, and the controller204 can control one or more of a shutter speed and an aperture todetermine how much light each detector in the light detector array 212receives. This determines, at least in part, the intensities of eachpixel in a resulting image. The controller 204 may also perform anevaluation of each image. For example, the controller 204 may determinethat an image is overexposed based on pixel data of the image. This mayinvolve determining that more than a threshold number of pixels in theoutput image are a maximum intensity value. The controller 204 mayadjust exposure parameters in order to achieve desired pixel statistics(e.g., a desired average pixel intensity value). In multi-channelimages, such as RGB images, the controller may perform the evaluationfor each channel. Further examples of auto exposure using the imagecapture device are provided below with respect to FIGS. 3-9.

Similarly, the controller 204 may use outputs from the one or moresensors 214 to control how images are captured. For example, a locationsensor (e.g., GPS), a time sensor (e.g., a clock), or a light sensor mayprovide a signal indicating an operational context of the image capturedevice 202 that causes the image capture device to adjustment to one ormore parameters for capturing images. For example, a time signal,location signal, or light signal may indicate an expected shift inlighting that prompts the image capture device 202 to change operationmodes. In other examples, the computing device 216 may prompt suchadjustments to parameters of the image capture device 202. For example,the computing device 216 may send a control signal to the controller 204indicating that the controller 204 should capture images in a differentmanner. For example, the computing device 216 may use images from theimage capture device 202 to perform object detection operations. Thecomputing device 216 may assign a confidence level indicative of thelikelihood that an object was accurately detected, and may promptchanges to auto exposure based on the confidence level. Other types ofcontrol signals from the computing device 216 are possible.

FIG. 3 is a block diagram of a simplified method of automatic exposureof an image, according to an example embodiment. In particular, FIG. 3shows a method 300 that can be performed by system 100, system 200, orcomponents thereof. For example, the image capture device 110 or theimage capture device 202, either alone or in combination with one ormore computing devices, can be used to carry out functions in method300.

At block 302, method 300 includes setting parameter values for capturingone or more images. For example, the parameter values may correspond toan aperture, shutter speed, and detector sensitivity (e.g., an ISOrating) of an image capture device.

At block 304, method 300 includes capturing one or more images using theimage capture device. In FIG. 3, a plurality of sequentially-relatedimages are shown, including image 306, image 308, and image 310. Someimages may be sampled from the plurality of images for purposes ofobject detection, edge detection, or other image processing techniques.While only some of the images might be sampled, each image can be usedfor adjusting parameter values of the image capture device. This allowsfor an image capture device to provide images that are more effectivefor image processing operations performed by the image capture device oranother computing device.

At block 312, method 300 includes evaluating the one or more images. Forexample, the image capture device or a separate computing device mayanalyze pixel statistics to determine if an image is overexposed (e.g.,by determining that an average pixel intensity exceeds a thresholdintensity.

Block 312 progresses again to block 302, such that parameter values foradditional images can be set. In this manner, the image capture devicecan re-adjust the exposure parameters adaptively as lightingcharacteristics of the environment change. Further details of adaptiveauto exposure are described below with respect to FIGS. 4-9.

III. Example Auto Exposure Scenario

FIGS. 4-8D show an example scenario in which a multi-channel image isexposed in accordance with a plurality of different auto exposureimplementations. The following examples show one context in which animage capture device may automatically switch operation modes, or inwhich a sensor or another computing device may provide a signal thatprompts the image capture device to switch operation modes. In thefollowing description, it should be understood that an “input image”refers to an image initially captured by an image capture device. Theinput image is used for determining exposure parameters for capturingone or more subsequent images. An “output image” refers to an imagecaptured using the exposure parameters determined from the input image.

FIG. 4 is an input image, according to an example embodiment. Inparticular, FIG. 4 shows an input image 400 captured using initialexposure parameters. FIG. 4 also shows a representation of average pixelintensities for each channel (red channel 402, green channel 404, andblue channel 406) of the input image 400. Each channel is represented bya bar that extends from a left to right in FIG. 4. A bar reaching theright edge of FIG. 4 indicates that the corresponding channel issaturated and overexposed. As shown in FIG. 4, the red channel 402 andthe green channel 404 have notably higher pixel intensities relative tothe blue channel 406. This may be due to lighting conditions in theenvironment. For example, sunrise or sunset can increase the amount ofred light in the environment, while limiting blue light. Thus, if theinput image 400 is captured using fixed white balance gains optimizedfor daylight, an image with imbalanced channels results. Other scenariosmight have similar effects on lighting, such as entering an area withprimarily artificial lighting, such as a roadway tunnel.

FIG. 4 also shows an object 408 in the environment. In this example, theobject 408 is a license plate, though other objects might be of interestin a given scene, such as vehicles, street signs, or other objects thatcan be detected using image processing techniques. Further detailsregarding the object 408 are provided below with respect to FIG. 8A.

FIG. 5 is an output image using non-weighted automatic exposure,according to an example embodiment. In particular, FIG. 5 shows a firstoutput image 500 captured using exposure parameters that are based on anon-weighted average of pixel intensities of the input image 400. FIG. 5also shows a representation of average pixel intensities for eachchannel (red channel 502, green channel 504, and blue channel 506) ofthe first output image 500.

In the example scenario, exposure parameters are set for the firstoutput image 500 that apply a gain relative to the input image 400 basedon a desired average pixel intensity. For example the desired averagepixel intensity may be a desired average of all pixels in the image(e.g., an intensity of 100 for a byte image ranging from 0 to 255brightness levels). The gain may be calculated as D*3/(R+G+B), where Dis the desired average pixel intensity, R is the average intensity ofthe red channel, G is the average intensity of the green channel, and Bis the average intensity of the blue channel.

In the input image 400, the average across all channels ((R+G+B)/3) isaround 71 out of 255, the desired average pixel intensity is 100 out of255, and a gain of about 1.41 results. As shown in FIG. 5, the redchannel 502 and the green channel 504 have notably higher pixelintensities relative to the blue channel 506. Further, the red channel502 and the green channel 504 are overexposed, resulting in at leastsome pixels being clipped in the first output image 500. Thisillustrates an example of how a non-weighted average may result inexposure parameters that overexpose one or more channels (the redchannel 502 and the green channel 504) when there is an outlier channelin the input image (the blue channel 406).

The average pixel intensities of red, blue, a green channels from theinput image 400 are reproduced for reference above the first outputimage 500. This illustrates a difference between the input image 400 andthe first output image 500 resulting from altering exposure parameters.As shown in FIG. 5, a gain was applied that resulted in overexposing thered channel 502 and the green channel 504.

FIG. 5 also shows an object 508 in the environment. Further detailsregarding the object 508 are provided below with respect to FIG. 8B.

FIG. 6 is an output image using channel-weighted automatic exposure,according to an example embodiment. In particular, FIG. 6 shows a secondoutput image 600 captured using exposure parameters that are based on achannel-weighted average of pixel intensities of the input image 400.FIG. 6 also shows a representation of average pixel intensities for eachchannel (red channel 602, green channel 604, and blue channel 606) ofthe second output image 600.

In the example scenario, exposure parameters are set for the secondoutput image 600 that apply a gain relative to the input image 400 basedon a desired average pixel intensity. For example the desired averagepixel intensity may be a desired average of all pixels in the image(e.g., a brightness of 100 for a byte image ranging from 0 to 255brightness levels). The gain may be calculated using a weighted averageof the channels. For example, the gain can be D/(x*R+y*G+z*B), where Dis the desired average pixel intensity, R is the average intensity ofthe red channel, G is the average intensity of the green channel, and Bis the average intensity of the blue channel. Weights x, y, and z can bedetermined in order to mitigate possible overexposure of one or morechannels. For example, a luma-equivalent set of weights (0.2126 for R,0.7152 for G, and 0.072 for B) can be used.

In the input image 400, the weighted average across all channels(0.2126*R+0.7152*G+0.072*B) is around 85 out of 255, the desired averagepixel intensity is 100 out of 255, and a gain of about 1.18 results.This is less than the example gain of 1.41 associated with thenon-weighted average of FIG. 5. As shown in FIG. 6, the red channel 602and the green channel 604 have notably higher pixel intensities relativeto the blue channel 606. However, unlike FIG. 5, the green channel 604is not overexposed in FIG. 6. This illustrates an example of how achannel-weighted average may result in exposure parameters that are lesslikely to overexpose channels than a non-weighted average, even inscenarios with an outlier channel in the input image (the blue channel406).

The average pixel intensities of red, blue, and green channels from theinput image 400 are reproduced for reference above the second outputimage 600. This illustrates a difference between the input image 400 andthe second output image 600 resulting from altering exposure parameters.As shown in FIG. 6, a gain was applied that resulted in overexposing thered channel 602. Accordingly, in contexts like that in FIG. 6 where onechannel (blue channel 606) is an outlier, different weights could beapplied to achieve an output image that is more effective for imageprocessing.

FIG. 6 also shows an object 608 in the environment. Further detailsregarding the object 608 are provided below with respect to FIG. 8C.

FIG. 7 is an output image using selective channel-weighted automaticexposure, according to an example embodiment. In particular, FIG. 7shows a third output image 700 captured using exposure parameters thatare based on a selective channel-weighted average of pixel intensitiesof the input image 400. FIG. 7 also shows a representation of averagepixel intensities for each channel (red channel 702, green channel 704,and blue channel 706) of the third output image 700.

In the example scenario, exposure parameters are set for the thirdoutput image 700 that apply a gain relative to the input image 400 basedon a desired average pixel intensity. For example the desired averagepixel intensity may be a desired average of all pixels in the image(e.g., a brightness of 100 for a byte image ranging from 0 to 255brightness levels). The gain may be calculated using a weighted averageof the channels other than an outlier channel. For example, FIG. 4 showsthat the blue channel 406 is an outlier relative to the red channel 402and the green channel 404. This can be determined based on one or morethresholds. For example, a difference in average pixel intensity can bedetermined between each channel, and if a given channel has an averagedifference above a threshold difference (e.g., greater than 30% of thedesired average pixel intensity), then the channel is considered anoutlier. Other thresholds can be used as well. In the present example,the average intensity for the blue channel 406 is 25, the averageintensity for the red channel 402 is 102, and the average intensity forthe green channel 404 is 85. The blue channel 406 has intensitydifferences of 77 and 60 relative to the red and blue channelsrespectively. Accordingly, the average difference for the blue channel406 is 68.5, making the blue channel an outlier. This may trigger theimage capture device to use a selective channel-weighted average fordetermining exposure parameters for the third output image 700.

In examples in which a channel is omitted from a channel-weightedaverage, the other weights are adjusted to output the gain. In theexample scenario of FIG. 7, gain be D/(x*R+y*G+0*B), where D is thedesired average pixel intensity, R is the average intensity of the redchannel, G is the average intensity of the green channel, and B is theaverage intensity of the blue channel. B is multiplied by 0 in order toremove the blue channel 406 from the weighted average. In an exampleusing luma-equivalent set of weights (0.2126 for R, 0.7152 for G, and0.072 for B), the weights are adjusted to account for removing theweights attributed to the blue channel by normalizing the sum of the setof weights. In the example scenario of FIG. 7, this is done by adjustingx from 0.2126 to 0.2486 and y is adjusted from to 0.71520 to 0.7512.This results in a gain of 1.12.

As shown in FIG. 7, the red channel 702 and the green channel 704 stillhave notably higher pixel intensities relative to the blue channel 706.However, in FIG. 7 the green channel 704 is less exposed than the greenchannel 604 in FIG. 6. Further, though the red channel 702 isoverexposed, fewer pixels are clipped in the third output image 700 thanin the second output image 600 because the gain for the third outputimage 700 (1.12) is less than that in the second output image 600(1.18). This illustrates an example of how a selective channel-weightedaverage may result in exposure parameters that are less likely tooverexpose channels than a non-weighted average or a weighted average,particularly in scenarios with an outlier channel in the input image(the blue channel 406).

The average pixel intensities of red, blue, a green channels from theinput image 400 are reproduced for reference above the third outputimage 700. This illustrates a difference between the input image 400 andthe third output image 700 resulting from altering exposure parameters.As shown in FIG. 7, a gain was applied that resulted in overexposing thered channel 702. Accordingly, in contexts like that in FIG. 7 where onechannel (blue channel 706) is an outlier, different weights (e.g.,non-luma equivalent weights) could be applied to achieve an output imagethat is more effective for image processing.

FIG. 7 also shows an object 708 in the environment. Further detailsregarding the object 708 are provided below with respect to FIG. 8D.

FIG. 8A is a portion 800 of the input image 400, according to an exampleembodiment. In particular, FIG. 8A shows a close-up view of the object408. The object 408 is depicted as a license plate, though other objectsare represented in the input image 400 and may be relevant depending oncontext. For example, the license plate is on a vehicle that may bedetected using an image for purposes of navigating a vehicle associatedwith the image capture device. FIG. 8A shows that the license plate inthe input image 400 is legible.

FIG. 8B is a portion of the output image using non-weighted automaticexposure, according to an example embodiment. In particular, FIG. 8Bshows a portion 802 of the first output image 500. FIG. 8B shows aclose-up view of the object 508, which is depicted as a license plate.The license plate in the first output image 500 is less legible than thelicense plate depicted in the input image 400. This decrease inlegibility is due to overexposure of the red channel 502 and the greenchannel 504 in the first output image 500. Accordingly, FIG. 8Billustrates how typical auto exposure operations (e.g., using anon-weighted average) for a multi-channel image may be less suitable incertain environments.

FIG. 8C is a portion of the output image using channel-weightedautomatic exposure, according to an example embodiment. In particular,FIG. 8C shows a portion 804 of the second output image 600. FIG. 8Cshows a close-up view of the object 608, which is depicted as a licenseplate. The license plate in the second output image 600 is as legibleas, or more legible than, the license plate depicted in the input image400. This is due to lessened overexposure of the red channel 602 and thegreen channel 604 in the second output image 600 relative to the firstoutput image 500. In addition, the average pixel intensity of the secondoutput image 600 is closer to the desired average pixel intensity of 100relative to the input image 400. Accordingly, FIG. 8C illustrates howusing a weighted average for auto exposure operations for amulti-channel image may be more suitable than a non-weighted average incertain environments.

FIG. 8D is a portion of the output image using selectivechannel-weighted automatic exposure, according to an example embodiment.In particular, FIG. 8D shows a portion 808 of the third output image700. FIG. 8D shows a close-up view of the object 708, which is depictedas a license plate. The license plate in the third output image 700 isas legible as, or more legible than, the license plate depicted in theinput image 400. This is due to lessened overexposure of the red channel702 and the green channel 704 in the second output image 600 relative tothe first output image 500 and the second output image 600. In addition,the average pixel intensity of the third output image 700 is closer tothe desired average pixel intensity of 100 relative to the input image400. Accordingly, FIG. 8D illustrates how using a selective weightedaverage for auto exposure operations for a multi-channel image may bemore suitable than a non-weighted average in certain environments.

FIGS. 8A-8D show an example scenario in which lighting conditions causeauto exposure operations of a first mode of an image capture device(e.g., a non-weighted average mode) to overexpose one or more channelsin an output image. In such a scenario, a second mode of the imagecapture device may be used to switch the auto exposure operations sothat they are better suited for the lighting conditions. For example,the second mode can use a channel-weighted average or selectivechannel-weighted average to determine a gain that decreases thelikelihood of overexposing one or more channels. Further detailsregarding these operations are described below with respect to FIG. 9.

III. Example Methods

FIG. 9 is a block diagram of a method, according to an exampleembodiment. In particular, FIG. 9 depicts a method 900 for use incapturing images using an image capture device. Method 900 may beimplemented in accordance with system 100, system 200, or thedescription thereof. For example, aspects of the functions of method 900may be performed by the system controller 102, the image capture device202, or by logical circuitry configured to implement the functionsdescribed above with respect to FIGS. 1-8D.

At block 902, method 900 includes capturing, by an image capture device,a first image comprising a plurality of pixels, wherein each pixelcomprises a plurality of channels, and wherein the first image iscaptured in accordance with first exposure parameters. For example, thefirst image can be an RBG image captured using a first combination ofaperture, shutter speed, and detector sensitivity for the image capturedevice. For example, the input image 400 depicted in FIG. 4 may be anexample of the first image.

At block 904, method 900 includes determining, by a controller of theimage capture device, average pixel intensities for each of theplurality of channels.

At block 906, method 900 includes determining, by the controller, aweighted average of pixel intensities using the average pixelintensities. For example, the weighted average of pixel intensitiescould correspond to luma-equivalent weights for an RGB image, or anotherset of weights. For example, block 906 may be performed in accordancewith the description of FIGS. 6 and 7.

At block 908, method 900 includes setting a gain, by the controller,that is proportional to a ratio of a desired average pixel intensityrelative to the weighted average of pixel intensities. For example, thegain may equal the desired average pixel intensity (e.g., 100 out of255) divided by the weighted average. For example, block 908 may beperformed in accordance with the description of FIGS. 6 and 7.

At block 910, method 900 includes setting, by the controller, secondexposure parameters for a second image. The second exposure parametersare based on the gain. For example, one or more of an aperture, shutterspeed, or detector sensitivity can be determined to provide thedetermined gain.

At block 912, method 900 includes capturing, by the image capturedevice, the second image in accordance with the second exposureparameters.

Within examples, method 900 further includes determining an operationalcondition associated with the image capture device. In these examples,determining the average pixel intensities (block 904), determining theweighted average (block 906), and setting the gain (block 908) areperformed responsive to determining the operational condition of theimage capture device. For example, the operational condition may includean expected shift in lighting of one or more of the channels. Forexample, this expected shift in lighting may correspond to a time ofday. For example, the time of day may be associated with sunrise orsunset on that day. In other examples, the image capture device iscoupled to a vehicle, and the expected shift in lighting corresponds toa location of the vehicle, such as a tunnel or another location withprimarily artificial lighting.

Within examples, method 900 further includes determining that at leasttwo of the plurality of channels are overexposed based on pixel datafrom the first image. For example, the first image may be an autoexposed image that has become overexposed due to lighting conditions inthe environment. In these examples, determining the average pixelintensities (block 904), determining the weighted average (block 906),and setting the gain (block 908) are performed responsive to determiningthat at least two of the plurality of channels are overexposed.

Within examples, the plurality of channels includes a red channel, agreen channel, and a blue channel, and determining the weighted averageincludes weighting the average pixel values for each channel using aluminosity function. For example, luma-equivalent weights can be usedfor each channel.

Within examples, determining the weighted average includes determiningan outlier channel from the plurality of channels, determining theweighted average based on channels other than the outlier channel. Inthese examples, determining the outlier channel may include determiningdifferences in average pixel intensities between each of the pluralityof channels and identifying the outlier channel based on the outlierchannel corresponding to a plurality of differences in average pixelintensities that exceed a threshold intensity difference. The thresholdintensity difference can be greater than or equal to 30% of a targetpixel intensity. For example, this may be performed in accordance withFIG. 7 and the description thereof.

Within examples, the second image immediately follows the first image ina sequence of images, and setting the second exposure parameters for thesecond image comprises adjusting the first exposure parameters to levelsthat correspond to a target pixel intensity based on the weightedaverage of pixel intensities.

Within examples, the first exposure parameters and the second exposureparameters include one or more of a shutter speed of the image capturedevice or a lens aperture of the image capture device.

Within examples, method 900 further includes identifying an object usingthe first image. In these examples, determining the average pixelintensities (block 904), determining the weighted average (block 906),and setting the gain (block 906) are performed based on identifying theobject. For example, identifying the object using the first image mayinclude identifying a first object (e.g., a vehicle) using the firstimage. Determining the average pixel intensities (block 904),determining the weighted average (block 906), and setting the gain(block 906) can be performed responsive to identifying the first objectin order to identify a related second object (e.g., a license plate onthe vehicle) based on the first image.

Though particular embodiments described herein describe RGB images,other multi-channel images are contemplated. Further, thoughluma-equivalent weights are described for a channel-weighted average,other sets of weights are possible, and may depend on an expected shiftin lighting within the environment. For example, the weights may beconfigured to lessen the relative intensity of one or more channels.Other examples of weights are possible.

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anillustrative embodiment may include elements that are not illustrated inthe Figures.

A step or block that represents a processing of information cancorrespond to circuitry that can be configured to perform the specificlogical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, aphysical computer (e.g., a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC)), or a portion of programcode (including related data). The program code can include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data can be stored on any type of computer readable medium suchas a storage device including a disk, hard drive, or other storagemedium.

The computer readable medium can also include non-transitory computerreadable media such as computer-readable media that store data for shortperiods of time like register memory, processor cache, and random accessmemory (RAM). The computer readable media can also includenon-transitory computer readable media that store program code and/ordata for longer periods of time. Thus, the computer readable media mayinclude secondary or persistent long term storage, like read only memory(ROM), optical or magnetic disks, compact-disc read only memory(CD-ROM), for example. The computer readable media can also be any othervolatile or non-volatile storage systems. A computer readable medium canbe considered a computer readable storage medium, for example, or atangible storage device.

While various examples and embodiments have been disclosed, otherexamples and embodiments will be apparent to those skilled in the art.The various disclosed examples and embodiments are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. An image capture device comprising: a lightdetector array; and a controller comprising: one or more processors; amemory; and program instructions stored in the memory and executable bythe one or more processors to perform functions comprising: using thelight detector array to capture a first image in accordance with firstexposure parameters, wherein the first image comprises a plurality ofpixels, and wherein each pixel comprises a plurality of channels;determining average pixel intensities for each of the plurality ofchannels; determining a weighted average of pixel intensities using theaverage pixel intensities; setting a gain based on the weighted averageof pixel intensities; setting second exposure parameters for a secondimage based on the gain; and using the light detector array to capturethe second image in accordance with the second exposure parameters. 2.The image capture device of claim 1, wherein the image capture deviceoperates in a first mode and a second mode, the functions furthercomprising: receiving a signal to switch from the first mode to thesecond mode, wherein determining the average pixel intensities,determining the weighted average, and setting the gain are performedresponsive to receiving a signal to switch from the first mode to thesecond mode.
 3. The image capture device of claim 2, wherein the imagecapture device is part of a system coupled to a vehicle, and wherein thesignal to switch from the first mode to the second mode is a controlsignal from a system controller of the system.
 4. The image capturedevice of claim 1, wherein the functions further comprise: determiningan operational condition associated with the image capture device,wherein determining the average pixel intensities, determining theweighted average, and setting the gain are performed responsive todetermining the operational condition.
 5. The image capture device ofclaim 4, wherein the operational condition comprises an expected shiftin lighting of one or more of the channels.
 6. The image capture deviceof claim 5, wherein the expected shift in lighting is based on at leastone of a time of day or a location.
 7. The image capture device of claim1, wherein the functions further comprise: determining that at least twoof the plurality of channels are overexposed based on pixel data fromthe first image, wherein determining the average pixel intensities,determining the weighted average, and setting the gain are performedresponsive to determining that at least two of the plurality of channelsare overexposed.
 8. The image capture device of claim 1, wherein theplurality of channels comprise a red channel, a green channel, and ablue channel, and wherein determining the weighted average comprisesweighting the average pixel values for each channel using a luminosityfunction.
 9. The image capture device of claim 1, wherein determiningthe weighted average comprises: determining an outlier channel from theplurality of channels; and determining the weighted average based onchannels other than the outlier channel.
 10. The image capture device ofclaim 9, wherein determining the outlier channel comprises: determiningdifferences in average pixel intensities between each of the pluralityof channels; and identifying the outlier channel based on the outlierchannel corresponding to a plurality of differences in average pixelintensities that exceed a threshold intensity difference.
 11. The imagecapture device of claim 1, wherein the first exposure parameters and thesecond exposure parameters comprise one or more of a shutter speed ofthe image capture device or a lens aperture of the image capture device.12. A method comprising: causing an image capture device to capture afirst image in accordance with first exposure parameters, wherein thefirst image comprises a plurality of pixels, and wherein each pixelcomprises a plurality of channels; determining average pixel intensitiesfor each of the plurality of channels; determining a weighted average ofpixel intensities using the average pixel intensities; setting a gainbased on the weighted average of pixel intensities; setting secondexposure parameters for a second image based on the gain; and causingthe image capture device to capture the second image in accordance withthe second exposure parameters.
 13. The method of claim 12, furthercomprising: determining an operational condition associated with theimage capture device, wherein determining the average pixel intensities,determining the weighted average, and setting the gain are performedresponsive to determining the operational condition.
 14. The method ofclaim 13, wherein the operational condition comprises an expected shiftin lighting of one or more of the channels.
 15. The method of claim 14,wherein the expected shift in lighting is based on at least one of atime of day or a location.
 16. The method of claim 12, furthercomprising: determining that at least two of the plurality of channelsare overexposed based on pixel data from the first image, whereindetermining the average pixel intensities, determining the weightedaverage, and setting the gain are performed responsive to determiningthat at least two of the plurality of channels are overexposed.
 17. Themethod of claim 16, wherein the plurality of channels comprise a redchannel, a green channel, and a blue channel, and wherein determiningthe weighted average comprises weighting the average pixel values foreach channel using a luminosity function.
 18. The method of claim 12,wherein determining the weighted average comprises: determining anoutlier channel from the plurality of channels; and determining theweighted average based on channels other than the outlier channel. 19.The method of claim 18, wherein determining the outlier channelcomprises determining differences in average pixel intensities betweeneach of the plurality of channels; identifying the outlier channel basedon the outlier channel corresponding to a plurality of differences inaverage pixel intensities that exceed a threshold intensity difference.20. The method of claim 12, wherein the first exposure parameters andthe second exposure parameters comprise one or more of a shutter speedof the image capture device or a lens aperture of the image capturedevice.
 21. The method of claim 12, further comprising: identifying anobject using the first image, wherein determining the average pixelintensities, determining the weighted average, and setting the gain areperformed based on identifying the object.
 22. A non-transitory computerreadable medium having instructions stored thereon, that when executedby one or more processors cause performance of functions comprising:causing an image capture device to capture a first image in accordancewith first exposure parameters, wherein the first image comprises aplurality of pixels, and wherein each pixel comprises a plurality ofchannels; determining average pixel intensities for each of theplurality of channels; determining a weighted average of pixelintensities using the average pixel intensities; setting a gain based onthe weighted average of pixel intensities; setting second exposureparameters for a second image based on the gain; and causing the imagecapture device to capture the second image in accordance with the secondexposure parameters.